A Method to Assess and Argue for Practical Significance in Software Engineering

A key goal of empirical research in software engineering is to assess practical significance, which answers the question whether observed effects some compared treatments show a relevant difference practice realistic scenarios. Even though plenty standard techniques exist statistical connecting it significance not straightforward or routinely done; indeed, only few studies principled and systematic way. In this paper, we argue that Bayesian data analysis provides suitable tools rigorously. We demonstrate our claims case study comparing different test techniques. The study's was previously analyzed (Afzal et al. , 2015) using focusing on significance. Here, build multilevel model same data, fit validate Our method apply cumulative prospect theory top quantitatively connect output practically meaningful context. This then basis both for assessing arguing demonstrates technically rigorous yet framework engineering. substantial side effect any uncertainty underlying will be propagated through model, its are made clear. Thus, combination with theory, supports seamlessly an context, thus potentially clarifying extending relevance practitioners.